(19) 



J 



Europaisches Patentamt 
European Patent Office 
Office europeen des brevets 



(12) 



(43) Date of publication: 

10.01.2001 Bulletin 2001/02 

(21 ) Application number 99305483.2 

(22) Date of filing: 09.07.1999 



(n) EP 1 067 757 A1 

EUROPEAN PATENT APPLICATION 

(51) lntCl7: H04N 1/047 



(84) Designated Contracting States: 


(72) Inventor Pilu, Maurizio 


AT BE CH CY DE DK ES Fl FR GB GR IE IT LI LU 


Bristol BS7 ORJ (GB) 


MC NL PT SE 




Designated Extension States: 


(74) Representative: Lawrence, Richard Anthony 


AL LT LV MK RO SI 


Hewlett-Packard Limited, 




IP Section, 


(71) Applicant: Hewlett-Packard Company 


Building 3, 


Palo Alto, California 94304-1112 (US) 


Filton Road 




Stoke Gifford, Bristol BS34 8QZ (GB) 



(54) Curled surface imaging system 

(57) The present invention relates to a system for 
de-warping images of an applicable surface, including 
applicable curie d surfaces, and in particular of images 
of curled documents (30). The system (1) includes a 
processor (25) linked to an image capture means (2) 
which: captures an image of the surface (30), said image 
having a warp corresponding to the non-planar surface; 



generates from the image a first set of data points rep- 
resenting the three-dimensional profile of the non-pla- 
nar surface relative (30) to a planar reference surface 
(12); fits to the first set of data points a second set of 
data points representative of an applicable mesh; and 
uses the second set of data points to texture-map the 
image (12) in order to de-warp the image. 
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Description 

[0001] The present invention relates to a system and a method for de-warping images o! an applicable surface, 
including developable curled surfaces, and in particular of images of curled documents. 

5 [0002] Images captured of a curled surface will in general exhibit distortions caused by image perspective, skew and 
compression or elongation caused by an uneven or a curled surface. Standard triangulation techniques can be used 
to calculate the surface profile from captured images of the surface. For example, a camera can be used with a struc- 
tured light pattern in a platenless document imaging system to capture the image of a page or of a bound book together 
with depth information that can be inferred from the light pattern. 

10 [0003] Desktop flat bed scanners are very common in office imaging applications. Although these are relatively in- 
expensive and work well, a disadvantage is that these invariably take up a significant amount of desk space, which is 
always at a premium. 

[0004] Digital camera products are becoming common in many areas of still and motion photography, and as a result 
are becoming ever less expensive. However such cameras are still used almost exclusively for photography of people 
15 or places, and have yet to be adapted for use in office imaging applications. One reason for this is that a document 
such as a piece of paper or an open book lying face up on a supporting surface is generally not flat, because the 
document is not held against a transparent platen as in a flatbed scanner. 

[0005] Documents may also not lie at a consistent angle to the camera. In the case of the book, the spine will then 
be skewed at a variable angle to the optical axis of the camera lens. 
20 [0006] Therefore, camera-based capture of a document poses the problem of distortion of the captured image due 
to image perspective, skew and compression or elongation introduced by the uneven surface and page curl of the 
sheet or bound book. 

[0007] Page curl is one of the biggest problems encountered when capturing a document with a camera. The curled 
part of the document renders poorly on screen and printers, presents shadows. It is also hard to do stitching and optical 
25 character recognition with such a "warped 8 image. 

[0008] Recovering or "de-warping" page curl is a difficult problem to solve in general. Methods that are known include 
inferring shapes from shading, from texture, from overall contours of lines of text. These methods have so far proved 
to be fragile and often require a significant amount of computer processing power. 

[0009] One approach to solve this problem is to use structured light to obtain depth information, such as the distance 
30 of the page from a plane at right angles to the optical axis of the camera. Such an approach is disclosed in patent 
document US 5,760,925, in which a document is supported on an underlying support surface with a cameras mounted 
on above and to one side of the support surface, and a light stripe projector mounted on an opposite side of the support 
surface. The light stripe projector projects a pair of light stripes onto the document. The light stripes are parallel for 
portions of the document the same height above a reference surface, which is taken to be the support surface. The 
35 document is oriented so that most of the curl is in the same direction as the light stripes, but because the document 
may not be flat in a transverse direction, the shape of the document is interpolated linearly between the light stripes. 
[001 0] This system can in principle capture an image of the document and correct this for page curl only when there 
is no curl transverse to the light stripes. Although more parallel tight stripes can in principle and at increased cost be 
added to gain additional curl information in the transverse direction, in practice this places a heavy burden on the 
40 available processing power and time available to capture and correct for document curl in a product that is commercially 
attractive in terms of cost and speed. 

[0011] It is an object of the present invention to address these issues. 

[001 2] Accordingly, the invention provides an imaging system for imaging a non-planar applicable surface, the system 
comprising a processor linked to an image capture means and being capable of: capturing at least one image of the 

45 surface, said image having a warp corresponding to the non-planar surface; and of generating therefrom a first set of 
data points representing the three-dimensional profile of the non-planar surface relative to a planar reference surface, 
wherein the processor is arranged to fit to the first set of data points a second set of data points representative of an 
applicable mesh and to use the second set of data points to texture-map the image in order to de-warp the image. 
[0013] Curied paper can be mathematically represented by an applicable or a developable surface that has the 

50 property of being isometric with the plane. In practical terms, this means that paper can be uncurled and/or unfolded 
to a plane without tearing. A set of measured three-dimensional data points representative of a curled applicable surface 
may consist of scattered and/or noisy data, in which case it is not possible to fit a general averaged surface such as 
a bicubic spline to the data and unroll, or "texture-map" the surface onto a plane without causing global distortions. 
The mesh is applicable before texture-mapping and therefore recovers at least to some extent the original applicable 

55 surface profile of the non-planar surface. In the case of a document imaging system, this permits the image of the 
document to be de-warped. 

[0014] Also according to the invention, there is provided a method of imaging a non-planar applicable surface using 
an imaging system comprising a processor linked to an image capture means, comprising the steps of: 
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i) capturing at least one image of the surface said image having a warp corresponding to the non-planar surface; 

ii) generating from the image a first set of data points representing the three-dimensional profile of the non-planar 
surface relative to a planar reference surface; 

5 

iii) fitting to the first set of data points a second set of data points representative of an applicable mesh; and 

iv) using the second set of data points to texture-map the image in order to de-warp the image. 

*0 [0015] In a preferred embodiment of the invention, in step iii) the mesh is distorted as the second set of data points 
is fit to the first set of data points to the extent that the mesh is no longer applicable, following which the distorted mesh 
is relaxed to an applicable state. This permits the mesh to average out deviations from an applicable state in the first 
set of data points, which can result in a better recovery of the original applicable surface from the scattered and/or 
noisy data. 

15 [0016] Preferably, prior to step iii) an "initial" surface is fit to the first set of data points, and in step iii) the mesh is fit 
to the initial surface. For example, the initial surface may be a bicubic spline surface. This in general wilt not be an 
applicable surface, in which case the mesh will become distorted as it is fit to the initial surface. 
[0017] However, in a preferred embodiment, the distortion of the mesh takes place in two stages so that the second 
set of data points may be better fit to the first set of data points. Here, the initial surface may be an applicable surface 

20 such as a plane, fit in a least squares routine to the first set of data points. The mesh is then not distorted in a first 
stage when it is fit to the initial surface, but rather in a second stage in which after fitting of the mesh to the initial surface 
at least some of the second set of data points are moved closer to corresponding ones of the first set of data points 
during which the mesh is distorted. 

[001 8] If some data points in the second set of data points do not correspond closely enough to any of the data points 
25 in the first set of data points, then these data points need not be fit the first set of data points. 

[0019] In a preferred embodiment of the invention, the relaxation of the mesh takes place in an iterative process in 
which the second set of data points is adjusted incrementally until distances between points in the second set of data 
points are equalized. 

[0020] The image capture means may'project a structured light pattern that forms separated light stripes across the 
30 non-planar applicable surface, the first set of data points being generated from the light stripes. Then, step ii) may 
include the steps of: 

a) creating a difference image by taking a difference between an image captured with the stripes and an image 
captured without the stripes; 

35 

b) thresholding the difference image to discard portions below a threshold; 

c) counting detected stripes across the difference image in order to identify individual stripes 

^0 d) triangulating the image of the non-planar surface at points corresponding with identified stripes to generate the 

first set of data points. 

[0021] The invention will now be described in further detail by way of example only, and with reference to the ac- 
companying drawings, in which: 

45 

Figure 1 is a schematic view in perspective of a document imaging system according to the invention, with a camera 
having a detector array mounted together with a light stripe projector on a post overlooking a document to be 
imaged; 

50 Figure 2 is a view of an open book as imaged by the camera; 

Figure 3 is a view of the open book with a light stripe pattern projected onto the book; 

Figure 4 is a schematic view in perspective showing a structured light pattern produced by the light stripe projector 
55 showing diverging sheets of light that bow concavely inwards toward a central planar light sheet; 

Figure 5 is a plot of light stripes formed by an intersection of the structured light pattern of Figure 3 with a plane 
transverse to the sheets; 
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Figure 6 is a plot of a polynomial fitted through five points taken from one of the light stripes of Figure 5; 

Figure 7 is a plot of a parametric light sheet cone constructed from polynomials such as those of Figure 6; 

5 Figure 8 is a flow chart depicting a calibration process for the roto-transiation between the camera and the fight 

stripe projector; 

Figures 9A and 9B show the error in detection of the profile of a planar surface using the document imaging system 
when an initial estimate of the roto-translation is used; 

10 

Figures 10A and 10B show the error in detection of the profile of a planar surface after calibration of the roto- 
translation according to the flow chart of Figure 8; 

Figure 11 A and 11 B show respectively the errors in the detection of a planar surface without and with radial distortion 
15 correction; 

Figure 12 shows the results of a stripe detection and labelling process; 

Figure 13 shows a light sheet and terminology used in a triangulation process to compute the intersection of the 
20 light sheet and the light stripe; 

Figure 14 shows a set of measured data points of an open book generated with the document scanning system; 

Figure 15 shows the profile of a surface fitted with a bicubic spline to the measured data points according to a prior 
25 art method of dewarp curl; 

Figure 16 show the prior art results of unrolling the bicubic spline onto a plane; 

Figure 1 7 is a schematic diagram of a prior art orthoimage method to dewarp curl when applied to the fitted surface; 

30 

Figure 18 is a schematic diagram of a triangular mesh used to approximate an applicable surface; 

Figure 19 is a schematic diagram of a way of estimating from the measured data points the approximate extent 
when de warped of a curled document; 

35 

Figures 20 A, 20B and 20C show in a two-dimensional analogy how the mesh is initially fit to the measured data 
points in a process which stretches the mesh so that it is no longer applicable; 

Figure 21 is a schematic diagram showing by analogy with a spring mesh how the mesh is relaxed back to an 
40 applicable state in which it is optimally fit to the measured data points; and 

Figure 22 shows a process of texture-mapping the relaxed mesh to de-warp the curled image of the document. 

[0022] Figure 1 shows a document imaging system 1 that has an electronic camera 2 a lower portion of which houses 
45 a light stripe projector 4 manufactured by Lasiris, Inc. of St. Laurent, Quebec, Canada, as model number 515L The 
camera 2 is mounted atop a support 6 that is clamped 8 to and rises above an edge 10 of a work surface 12. The 
camera 2 has a main lens 14 with an optical axis 16 that is directed across and down upon the work surface 12. The 
lens 14 has a field of view 18 that images an area 20 of the work surface 12 onto a two-dimensional CCD detector 
array 22 within the camera 2. 

so [0023] The detector array is connected 23 to a processor unit 25, which may, for example, be a personal computer 
with an expansion card for controlling the camera 2, light stripe projector 4, and for receiving and processing data 
received from the detector array 22. 

[0024] Ideally, the area 20 is at least of A4 document size. Similarly the light stripe projector 4 has a projecting lens 
24 that projects a structured light pattern 26 onto a work surface area 28 that is roughly coincident with the imaged 
55 area 20. The structured light pattern will be described in more detail below, but extends around a central axis 29 that 
is roughly coincident on the work surface 12 with the camera lens axis 16. The spread of the structured light pattern 
is sufficient to cover an A4-size area at about 300 mm distance. 

[0025] A document 30 has been placed within the area 20,28 defined by the camera lens 14 and structured light 
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pattern 26. The document is supported by the work surface 1 2 in a generally horizontal orientation, but is slightly curled. 
An image captured by the detector array 22 will therefore have perspective foreshortening owing to the oblique angle 
between the camera optical axis 16 and the document 30, and well as warp distortion due to the document curl. 
[0026] Such warp distortion can be seen in Figure 2, which illustrates an image 31 of an open book 32 as formed 
5 on the detector array 22 by the camera 2. The amount of warp distortion is greatest near the spine 34. 

[0027] Figure 3 shows an image 33 of the open book 32 when the structured light pattern 26 is projected towards 
the book 32 to produce fifteen separated light stripes 35 over the book 32. The book is oriented so that the light stripes 
35 are transverse to the spine 34. 

[0028] A difference can then be taken between the image 33 with the light stripe pattern 35 and the same image 31 
10 without the light stripe pattern 35, in order to detect the light stripes. 

[0029] As can be appreciated from Figures 1 and 3, the advantage of having the light stripe projector mounted 
together with and below the camera is that the furthest stripe 36 will always be in view of the camera, even if the stripe 
is projected beyond the further edge of a book. 

[0030] Figure 4 shows how the structured light pattern 26 is produced. A fixed 7 mW laser 38 projects a beam of 
15 light 40 to a first optical diffractive element 42 that diffracts the beam 40 into a vertically oriented strip 44. The vertical 
stripe 44 is then diffracted by a second optical diffractive element 46 into the structured light pattern 26 consisting of 
fifteen diverging, separate and non-intersecting sheets of light 48. 

[0031] The structured light pattern 26 is projected onto the document 30 with the projection axis 29 offset at an angle 
50 to permit triangulation of the light stripes 35 to characterise document curl. 
20 [0032] These diffractive elements 42,44 produce a set of seven conical light sheets 51,53 either side of a central 
planar light sheet 52. The central planar light sheet 52 contains a median ray 49, which also lies on the light stripe 
projector axis 29. 

[0033] Each set of light sheets 51,53 bows concavely inwards towards the central planar light sheet 52, with the 
result that the divergence between adjacent light sheets is a minimum at the middle of the light sheets 48. The sheets 
25 are symmetric about a plane that is transverse to the planar sheet and which comprises a median ray of the planar 
sheet 52. 

[0034] As shown in Figure 5, the conic light sheets 51,53 will in general generate curved, non-parallel light stripes 
on the document, with a concentration of light stripes along a line centrally transverse to the light stripes 35. The 
concentration of stripes corresponds with the minimum divergence between adjacent light sheets. In Figure 3, this 
30 concentration is about the book spine 34. In this example, the concentration of light stripes about the spine 34 will 
provide enhanced curl information in the region of greatest document curl. 

[0035] Triangulation of conic light sheets is, however, a non-trivial problem. For this reason a closed-form solution 
to this triangulation problem is described below that can be applied in general with this kind of structured light to 
characterise document curl. The closed form of the triangulation also allows the use of a standard optimisation method 

35 to perform an initial.calibration of the camera 2 and light stripe projector 4 system. 

[0036] Although the use of a multiple line structured light pattern has advantages in terms of cost, the time needed 
to capture an image, and mechanical complexity over traditional laser scanning methods in the sense that there are 
no moving parts such as galvanometers or stepper motors, there is a drawback in that the three-dimensional resolution 
is less, being limited to the number of lines in one direction. This drawback is partly mitigated by the concentration of 

40 lines in the region of greatest curl and, as will be explained below, by the use of novel methods to characterise and 
de-warp image curt. 

[0037] Triangulation works as follows. First, light stripes 48 are projected onto an object which is viewed by and 
projected onto the camera image plane at the detector array 22. Let us suppose for the moment that the laser projects 
just a single light sheet of a known shape and defined by a corresponding known equation in the camera reference 

45 system, which when cast onto an object and imaged by the camera produces a single curve (or stripe) on the image 
plane. A given point of the stripe defines a line in space going from the camera optical axis through the image. The 
intersection between the light sheet and this line defines a three-dimensional point in the camera reference system 
that is on the surface of the object. By repeating the procedure for each stripe point, we can effectively recover all the 
object's points that lie on the curve defined by the intersection of the projected light sheet and the object surface. 

50 [0038] In this invention we do not have a single light sheet but rather a set of them slightly displaced in order to cover 
a larger portion of the object, and thus obtain a three-dimensional snapshot of it. In the single light sheet case we knew 
that any image stripe point corresponded to the projection of a 3D point of a known stripe and this is why it is possible 
to do triangulation unambiguously. Conversely, in the multiple light sheet case we do not actually know which particular 
light sheet generated that projection and so some sort of stripe labelling or identification is necessary to determine 

55 which image stripe was generated by a particular light sheet. 

[0039] The camera 2 and light stripe generator 4 system is initially calibrated by measuring a reference surface, 
which for convenience may be a support surface 12. The vertical displacement between the undistorted projection on 
the reference surface and the distorted projection on the curled document is a unique function of depth or height of 
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the curled document relative to the reference surface. 

[0040] The diffractive light stripe projector 4 produces a structured light pattern that with stripes 48 each of which 
that has a periodic intensity variation along its length. To a first approximation, the peaks in light intensity of the structured 
light pattern therefore occur at points, which on a spherical surface centered on the light stripe projector can be rep- 
s resented by the following equations: 



10 



and 



where 



20 



25 



30 



35 



N u = 



(1) 



and where (x t y) = (0,0) is on the projection axis 29, D is the distance from the light stripe projector 4, X is the laser 
wavelength, A-j is the period of the grating for diffractive element 42 and A 2 is the period of the grating for diffractive 
element 46. 

[0041] Figure 5 shows fifteen light stripes formed by the intersection of a plane spaced 0.5 m from the light stripe 
projector and at right angles to the light sheet projection axis 29. The central planar light sheet 52 produces a straight 
light stripe 54, and light stripes 55,57 on either side of the central light stripe 54 bend inwards towards the central light 
stripe 54. The light stripes are therefore concentrated along a central line 56 transverse to the central stripe 54. 
[0042] In order to perform triangulation in closed-form on the projected light stripe pattern, it is necessary to express 
this pattern and hence each light sheet 48 in a mathematical form. Therefore, five points 58 which correspond to 
subsidiary maxima along each light stripe 50 are used, as shown in Figure 6, to generate a second-order polynomial 
of the projected stripe on the orthogonal plane at a given distance. Although the stripe is, strictly speaking, not quadratic, 
we have noticed that the deviation from the data is less than 0.01% when the polynomial is of second order. The 
equation of the polynomial r can be expressed in parametric form as: 



40 



45 



z r.N ~ c n 



50 



where the index N stands for the stripe number and a is a free parameter illustrated graphically in Figure 7. From this 
we can construct a cone 70 centered on the projection axis 29, by letting vbe a parameter sweeping the cone length. 
The cone 70 is expressed as: 



55 



x 6ji ~ ^rjv = vc n 



(2) 
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[0043] As explained below, of particular interest is the algebraic form of each cone of light, which is obtained by 
elimination: 

[0044] In order to perform triangulation in closed-form, it is necessary also to know the relative orientation of the 
camera lens axis 16 and the light stripe projector axis 29, referred to herein as the roto-translation R 0L between the 

10 camera 2 and the light stripe projector 4. 

[0045] The intrinsic camera model employed in this invention is described by a conventional set of five parameters, 
which are the focal length f, the number of pixels per meter in the horizontal and vertical direction and ay, the 
"piercing point" (xq y 0 ) (assumed to be at the image centre) plus the radial distortion parameter K. 
[0046] The calibration of the camera parameters i y a* and ay, the "piercing point" (xo,y 0 ) can be estimated with a 

is method described by Tsai, R. Y., IEEE Transactions on Robotics and Automation, No. 4 pp. 323-344, 1987. 

[0047] The estimation of the roto-translation R OL is accomplished by a custom method based on an optimization 
process starting with the capture of sparse three-dimensional data of a planar object. An initial rough estimate of R OL 
is then determined. Following this, an iterative process shown in Figure 8 is used to adjust six parameters representative 
of R 0L (three Euler rotations angles and three translations) until triangulated data points become effectively planar. 

20 Minimization of errors is carried out by an implementation of the Levenberg-Marquard method. 

[0048] Figures 9A and 9B show two graphs that illustrate the errors in measuring a planar surface using the initial 
rough estimate of R OL . Figu res 1 0A and 1 0B show two similar graphs using the final estimated R OL after the optimisation 
process of Figure 8. These graphs show that the reduction in the standard deviation of the error in the measurement 
of the plane is reduced from 20 mm to less than 1 mm. The residual error is due to measurement noise. 

25 [0049] Correction of radial distortion is generally neglected in the field of document imaging. However, it has been 
discovered that this correction is important in order to obtain sufficiently accurate results. The mapping from distorted 
to undistorted co-ordinates is: 



30 

x = x + xK.d 2 i : 

y = y + yK t d 2 

35 

[0050] For simplicity of presentation, these new coordinates will in the following description be treated as the actual 
image coordinates, although one has to bear in mind that these are corrected coordinates derived from the above 
mapping. 

[0051] The camera 2 used in the present example has a radial distortion parameter K= O.OO^pixels/mm 2 . Figure 
40 11 A shows how, even when the R OL has been calibrated, if the radial distortion is not accounted for the error becomes 
very large. Once this distortion is allowed for, the distortion is as shown in Figure 11 B. 

[0052] Because there is more than one light stripe, it is necessary to identify each light detected stripe before trian- 
gulation is performed. There are two distinct parts in this process, the first one being stripe detection and the second 
one stripe labelling. 

45 [0053] The three-dimensional document image capture can be done by briefly flashing the laser partem and syn- 
chronously detecting with the detector array 22 the document image including the light stripe pattern, as shown in 
Figure 3. Either before or after this, the document is imaged without the light stripe pattern, as shown in Figure 2. There 
will then be two overlapping images, one with the pattern and one without and thus it is straightforward to use image 
differencing to make the stripes stand out. 

so [0054] However, the intensity value across stripes will in general be uneven, for example owing to subsidiary peaks 
as in equation (1) above, or because of uneven ambient illumination or paper reflectance. Therefore, the image of the 
lights is processed. Given the prevalently horizontal lines, the first step is to use a one-dimensional Laplacian operator 
(second derivative) applied only on the y (vertical) direction. The application of this operator gives the centre of the 
stripe a high negative value. This value can then be thresholded to obtain a binary image. The process is robust and 

55 fast but the use of a single threshold may inevitably cause some gaps in the continuity of the detected stripes, as shown 
in Figure 12. 

[0055] The method used to identify light stripes is as follows. First, the stripes are thinned down to one pixel thickness 
and connected pixels are joined together into a string. Next, strings that are too short are removed from the classification 
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as deemed noise. The result is a data set of pixels as shown in Figure 12, where string segments 80 are interspersed 
with gaps 82. 

[0056] Then for each string, a heuristic "strength - measure is computed as: 

5 

S=0.5*Length +0.5*Abs(Avg(Top 30% ofLapiacian value)) 

[0057] This is an equally weighted sum of the length and the average of the top third of the absolute value of the 
Laplacian values. We do not average all the values of the Laplacian along a string because the stripe intensity is not 

10 uniformly distributed and some faint sections might adversely affect the average. 

[0058] Next, for each column of pixels and starting from the top row of pixels of the image, we assign successive, 
increasing label numbers to, and only to, the strongest stripe points in the sense above. The numbering of stripes stops 
at the maximum number expected, here fifteen. Finally, for each string we assign a label equal to the most popular 
label assigned to all the points of that string. Figure 12 shows the labelling result in which all stripes are correctly 

15 identified. 

[0059] This approach, which is essentially a voting method, is very robust in general situations and can smoothly 
cope with gaps. It is also relatively fast to run with inexpensive hardware. It has the advantage that the light stripes are 
individually indistinguishable, with individual stripes being identified by counting the series of stripes. 
[0060] There are situations in which the method would fail to label stripes property, namely when the assumptions 

20 onto which it is based are not meet. For instance if a stripe is completely or largely invisible or occluded by, for example, 
a thick book edge, the label assignment will be meaningless. Therefore, in an alternative embodiment not illustrated 
in the drawings, the light stripes are made individually distinguishable by spatial modulation. 
[0061] Having said so, these are situation that should not occur in practice when the light stripe projector is property 
arranged with respect to the camera, for example being mounted on the same side of the document as the camera 

25 and below the level of the camera. 

[0062] The proposed approaches to identifying stripes are quick and simple compared with other approaches in 
which the stripes are temporally modulated or otherwise made individually distinguishable, for example by colour cod- 
ing. 

[0063] Three-dimensional data points can then be obtained via triangulation, which as illustrated in Figure 1 3 consists 
30 of finding the intersection between the sheet of light 48 and an optic ray going 84 through a given point 86 on the 
projected stripe 88 and a corresponding point 90 on the detected image 92 in the detector plane 94. 
[0064] Referring to Figure 1 3, let °P = (X, Y t Z) be a three-dimensional point in the camera reference system O, °p = 
(x,y) a stripe point in the image plane, L Z the conical surface representing the conical light sheet 48 in the light stripe 
projector reference system L, and R OL the transformation between the two reference systems expressed by the four- 
35 by-four matrix: 
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[0065] The triangulation problem is to finding the intersection between a generic elliptic cone and a line in space. 
First we transform the cone into the reference system of the camera via R OL by expressing a cone point in terms of a 
point in the O reference system transformed into L: 



50 
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'I: 



L . y z- z a N c N x = 0 



L y-* y 'x+o y 'y+o y m i+p y 



[0066] The parametric form of the optic ray is: 
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°z=tf 



5 where f is the focal length and the x and y are expressed in image coordinates. Then we write down a system that 
expresses the intersection between this cone and the optic ray: 



10 



15 



20 



25 



°x = tx, 

°y=iy i 

'z=tf 



[0067] By simple substitution, we arrive at a second order equation in the parameter of the optic ray t 

A? + Bt+C = 0^>[t v t 2 ] 

whose solutions t 1 and t 2 represent the two intersections of the ray with the cone. This equation can be solved analyt- 
ically and the rather knotty solution has been found but is omitted here for clarity. 

[0068] We are interested in only one of the above-mentioned intersections which turns out to be, because of the way 
we constructed the cone, the one corresponding to the smallest parameter u spanning on the half cone closer to the 
Z axis of the reference system L 

[0069] Hence, we transform both solutions back to the light stripe projector reference system L: 



30 



35 



and use the 2nd of Equations 2 to recover the two corresponding u's, that is: 
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( L Xc\ 
t x such that X =argmin|^ = L ^ N , X = 1,2 



45 [0070] Finally, the three-dimensional intersection point is given by: 



X=tx. t 



50 



Y=t Yi 



55 



Z=tf 

[0071] We have now found the coordinates of the point in space that belongs to the intersection of the light sheet 
with the object and whose projection is a particular stripe pixel in the image. 

[0072] This process has to be repeated for each pixel (and optionally at sub-pixel resolution) of each of the fifteen 



9 



EP 1 067 757 A1 



stripes. The triangutation process is very fast but if necessary it would be possible to sub-sample along each line. The 
result is a "cloud" of three-dimensional data points such as that shown in Figure 14. 

[0073] Now we have a cloud of three-dimensional points 95 representing the paper surface. The problem is how to 
use these data points 95 to undo or "de-warp" the curl distortion. 
5 [0074] It is in general difficult to de-warp an image of a curled document. The main problem is that paper is not a 
generic surface, but a "developable" surface, that is one that can be unfolded without tearing or stretching onto a plane. 
The Gaussian curvature K of a developable surface S(u.v) is zero everywhere, i.e. K(u,v) = 0. An applicable surface 
is a more general case of a developable surface, in which there are sharp creases. 

[0075] The conventional surface reconstruction approach of fitting and regularizing a surface, possibly with some 
10 discontinuities, does not apply to our problem since we not only need to reconstruct, but we also have to unfold this 
surface onto a plane. This is not possible if the reconstructed surface is not applicable in the first place. Hence, it is 
necessary to constrain the fitted surface to be applicable, that is, with zero Gaussian curvature everywhere, which is 
a not trivial operation. 

[0076] Figures 15 and 16 illustrate why a simple approach will in general not work. The three-dimensional data of 
15 Figure 1 4 has been smoothed and a bi-cubic spline surface 96 has been fitted. In the ideal case where data is noiseless 
and the light stripe projector and camera system is perfectly calibrated, a fitted surface should also be applicable, but 
in reality the surface we obtain is clearly not so. For example, see the little bumps in some places 98. 
If we now uncurl the page, we have to texture-map patches from the original image onto patches of a plane, a mapping 
computed by integration of finite differences in the meshed surface 96 as shown in Figure 16. 
20 [0077] However, by definition, a non-applicable surface can only be unrolled onto a plane by either tearing or stretch- 
ing which causes unnatural distortions in the unrolled document 100. This is due to the integrative nature of unrolling 
a surface where locally small errors tend to build up and lead to unsightly distortions. Figure 16 shows the distortion 
in the texture, which his caused by the irregularities in the farther side of the reconstructed planar mesh in Figure 15. 
[0078] So the problem of unrolling the page can be restated as a problem of fitting an applicable surface onto noisy 
25 data. 

[0079] A second problem is that the light stripes do not cover the entire page or there might be gaps right near the 
edges of the page/book. In this case we do not have three-dimensional data so we would not know how to unroll these 
regions. 

[0080] Briefly, the method used with the present invention uses a finite element model represented as a triangular 
30 mesh that simultaneously fits to the data and constrains the data to be isometric with a plane (i.e. applicable) by a 
relaxation process. 

[0081] First consider the problem in two-dimensions as illustrated in Figure 17. Here there is a first set of points 102 
representing noisy measurements of a curve along its length. A second set of points 104 can then be fit with a least 
squares fit to the first set 102. A connected piecewise linear curve 106 can be constructed going through the second 

35 set of points 104. The second set of points 104 can always be "undone" to a line 108, as the linear curve is isometric 
to a line. This property explains why many methods that seek to undo the page curl of books use a one-dimensional 
model of the curl and produce good results when the document curl is essentially cylindrical. However, in a general 
three-dimensional case it is not possible simply to unfold a two-dimensional set of data points representing a curled 
document owing to noise or other inaccuracies in the data points. 

40 [0082] There is a old technique used in cartography called orthoimage projection which is described here for its 
relevance to this work. The method essentially does not correct for page curl but simply projects the texture ortho- 
graphically onto a plane. This method, albeit simple and not prone to local distortions, is fundamentally flawed, because 
it does not unfold the document but rather just "pushes it down. 

[0083] A surface is called a developable surface when its Gaussian curvature vanishes at every point. Developable 
45 surfaces can be flattened onto a plane without stretching or tearing. Similarly, a developable surface is one that is 
obtained by bending a plane, where with bending we mean a transformation that preserves arc length. 
[0084] Note that not all ruled surfaces are developable. Developable surfaces are a special subclass of ruled sur- 
faces, that is surfaces that are generated by a straight line moving in space. 

[0085] A developable surface is a subclass of the more general applicable surfaces. In practice, applicable surfaces 
50 have the same properties as developable surfaces, but may have creases. 

[0086] The analytic form of a developable surface is a parametric equation of a ruled surface with the constraint of 
the tangent plane being the same along each ruling. This definition is per se impractical and is mainly suitable for 
interactive modelling or display. 

[0087] A Finite Element Model (FEM) can be used to represent an applicable or a developable surface, for example 
55 a mesh 110 such as that shown in Figure 18 having triangular tiles 111. Such a mesh can be deformed 112 to approx- 
imate 114 a developable or an applicable surface. When the mesh 100 is deformed, the tiles 111 remain unchanged. 
[0088] A developable or applicable surface can be modelled with a triangular mesh by assuming that the lengths of 
mesh edges 116 between mesh nodes 117 keeps constant as the mesh is deformed. Of course, making the mesh finer 
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can make any approximation error arbitrarily small. It has to be noted, however, that it is in general not possible to split 
triangles and refine the mesh locally to reduce the error in, say, regions of high curvature once the mesh has started 
deforming. 

[0089] Creases too can be modelled with such a deformable wire-frame mesh. In fact, by increasing the resolution 

5 of the mesh it is possible to model more and more accurately any applicable surface. 

[0090] The document curl characterisation process described above will in general produce noisy, sparse data as 
shown in Figure 14. The extent of the surface may not be known. Figure 19 shows one way to estimate the extent of 
the surface. A convex hull or rectangle 118 (or equivalently a square) enclosing all the data points 95 is projected onto 
the support plane 12. A rectangle 124 can then be deduced from extreme points of the projected lines 126,128. In 

w Figure 19, a B-spline fits the data 95 and estimates its extent by integration along some chosen curves 120,122. 

[0091] Alternatively, the document scanning system may permit a user to select the size of the document. Alterna- 
tively, the extent could be determined straight away from the image only using the known system geometry, that is, 
stand position with respect to camera 2 and camera calibration parameters. This latter approach would also help 
overcome problems of mismatch between the structured light pattern and the document dimensions. 

15 [0092] Such a mismatch could occur if part of the part of the three-dimensional data 95 does not belong to the same 
developable surface. This might be the case if the data is of a thick book or a small curled document and the structured 
light pattern is bigger than the document region. In this case there needs to be a way to tell what belongs to the 
document and what is does not. This could be done by analyzing a generic surface fitted to the data with a search for 
steep curvature or depth changes. Data points outside such sudden changes could then be discarded. Another way 

20 would be to allow for the mesh to break during the "relaxation" process described below wherever the process does 
not converge. 

[0093] Once the extent of the surface 118 and the corresponding planar projection 124 are known, the mesh 110 is 
fit to the noisy set of data points 95. The process can be understood with reference to Figures 20A, 20B and 20C, 
which show for clarity an "initialization" process in a two-dimensional analogy. 

25 [0094] First an "initial" surface, here a plane 130, is fit with a least squares deviation through the noisy three-dimen- 
sional set of data points 95. Then the planar mesh 110 is rotated and translated so as to coincide with this plane 130 
and the estimated extent 124 of the surface 118. Then each mesh node 117 is vertically translated 132 at right angles 
to the least squares fit plane 130 towards the closest data point 95. If there is no data point 95 within a certain radius, 
here taken to be one-third the distance to the nearest neighbouring node 117, then the node is left in place, as is the 

30 case in Figure 20C for one node 134. The result is a distorted mesh 133. 

[0095] At this stage, the mesh 1 33 is no longer applicable, that is the isometry with the plane 1 30 we started with is 
lost. However, albeit coarsely, the mesh 133 does now approximate the surface 118. The next stage is to adjust the 
mesh 113 so that it is again applicable, and this is done in a "relaxation" process. 
[0096] Let us first define the terminology to be used. Let 

35 

40 be a mesh node defined as a vector of corrdinates in a Cartesian system, and let 

X = {*1> 

45 be the set of nodes of the mesh. Also let e ip be an edge of the mesh joining two nodes x,- and x y and let 




so 

be the set of all the edges of the mesh. The mesh can then be represented by M = {X,£}. Let us also define a neigh- 
bourhood of a node x,as the set of nodes 



[0097] We shall indicate with dg = J(x i - x y ) r (x / - x y ) the Euclidean distance between two nodes and with o^-the ref- 
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erence distance that the mesh had in its original flat state. 

[0098] In order to transform the mesh to an applicable state while still approximating the data, an optimization method 
is used to minimize the deviation of the mesh from the ideal applicable state. Figure 21 illustrates byway of a, mechanical 
analogy a mesh 140 of springs 142. In a relaxed state, the mesh 140 has relaxed springs 142 of extension ^ connected 
to each other at nodes 144. Try? reticular structure 140 will therefore be at a minimum energy in a stable state when 
all the springs have extension c^and when this happens the mesh 140 is isometric with the plane. 
[0099] Hence, the problem is equivalent to that of minimizing the total elastic energy of the system: 



[0100] This is done using the well-known gradient descent methoc that iteratively adjusts the position of the nodes 
until the final, lowest energy is reached. Note that the elastic constant Kcan be ignored during the minimization process. 
[0101] Node co-ordinates are updated according to the followinc rule: 

f+1 _ / dU 



f+t t BU 
y. -y + w, -z~ 

where w is a factor that will be discussed later. 

[0102] Convergence is reached when ail the displacements fall below a set treshold. 
[0103] The derivatives are straightforward to compute and are: 




[0104] Note that these derivatives could be also rewritten as the resultant of the forces exerted to each one of the 
nodes 144 by all the springs 142 connected each particular node. 

[0105] Regarding the convergence properties of the iterative optimization procedure, it can be shown that conver- 
gence is achieved when 



W "~¥U7~ 
A, 

(similarly for y) and 0<w<2. 

[0106] A fitting experiment has confirmed this. 

[0107] The relaxation process described above behaves well and has been shown to approximate the surface very 
precisely. This is somewhat surprising, because the set of data points 95 is not used during the mesh relaxation. The 
basis for this surprising result lies in the highly constrained nature of a developable surface or its discrete approximation 
such as the mesh 110 with the contstraint that for each node /, dg =const When the mesh is initialized onto the data, 
the nodes do not satify this constraint. However, the relaxation procedure causes nodes to be displaced orthogonally 
to satisfy the contraints. The form of the surface does not change drammatically, which would be the case if the dis- 
placements were tangent. This key observation is what makes the relaxing mesh approximate the surface without data. 
[0108] Once the mesh is fitted properly to the three-dimensional data, the next phase is to texture-map the initial 
planar mesh. As we mentioned before, with this technique there is no need to unroll the surface just fitted, because 
we already have it to start with. 

[0109] Texture-mapping to de-warp the curled document consists of three phases, which are illustrated in Figure 22. 
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First, all tiles 111 in the planar mesh 110 are initialized and relaxed 150 to the characterised document surface 152 
such as to keep isometry. Using the known imaging geometry, the tile 111, which now lies on the three-dimensional 
surface 1 52, is back-projected 1 54 to the image plane 1 56 so as to obtain the texture 1 58 from the image that correspond 
to it. The final phase is to warp 160 the tile texture back 158 to its corresponding planar tile 111 so as to restore the 

5 texture as if it had been captured originally in a flat state. 

[0110] The warp stage 160 is a standard process and a number of excellent algorithms are described by George 
Wolberg in an book titled Digital Image Warping, published by IEEE Computer Society Press, 1991. 
[0111] The document curl characterisation approach described above works well regardless of the paper type and 
document complexity. The invention provides a practical and cheap means of characterising and de-warping page curl. 

10 in particular, the profile of the imaged document is determined by projecting a known two-dimensional structured light 
pattern and triangulating with the image of the pattern taken by the camera. The use of a two-dimensional pattern, as 
opposed to a single stripe or point, is particularly desirable in this applications because it does not require expensive 
moving parts (and their drivers) and allows quick characterisation of the page-curl in a single shot, not by sweeping a 
single beam over the page. 

15 [0112] In this example, the stripes are identified only from images of the plain stripes, without moving either the 
detector array or the lights stripe projector or imposing any kind of temporal or spatial modulation, which would increase 
heavily on the system cost. 

[0113] The method described above for characterising document curl in a platenless document scanning system is 
practical and fast and can be implemented with relatively inexpensive hardware for a document imaging application. 

20 [01 1 4] The document curl correction method presented above uses a mathematical model of paper, and an effective 
initialization and relaxation process for fitting this model to the data in a way that naturally produces an undistorted 
image. This is accomplished despite the fact that there are a large number of degrees of freedom and a large number 
of constraints that need to be satisfied with sparse and noisy data. This method has the ability to interpolate, extrapolate 
and self-complete wherever data is missing. The method produces high quality de-warped images of curled documents 

25 by modelling paper deformation in a physically realistic way. 



Claims 

30 1. An imaging system (1) for imaging a non-planar applicable surface (30), the system (1) comprising a processor 
(25) linked to an image capture means (2) and being capable of: capturing at least one image (31 ,33) of the surface 
(30), said image (31,33) having a warp corresponding to the non-planar surface; and of generating therefrom a 
first set of data points (95) representing the three-dimensional profile of the non-planar surface relative to a planar 
reference surface (12), wherein the processor .(25) is arranged to fit to the first set of data points (95) a second 

35 set of data points (117) representative of an applicable mesh (110) and to use the second set of data points (117) 

to texture-map the image (150,154,160) in order to de-warp the image (32). 

2. A method of imaging a non-planar applicable surface (30) using an imaging system (1) comprising a processor 
(25) linked to an image capture means (2), comprising the steps of: 

40 

i) capturing at least one image (31 ,33) of the surface (30) said image having a warp corresponding to the non- 
planar surface (30) ; 

ii) generating from the image (31,33) a first set of data points (95) representing the three-dimensional profile 
45 of the non-planar surface (30) relative to a planar reference surface (12); 

iii) fitting to the first set of data points a second set of data points (117) representative of an applicable mesh 
(110); and 

50 iv) using the second set of data points (117) to texture-map (150,154,160) the image (31,33) in order to de- 

warp the image. 

3. A method as claimed in Claim 2, in which in step iii) the mesh (133) is distorted as the second set of data points 
(117) is fit to the first set of data points (95) to the extent that the mesh (133) is no longer applicable, following 

55 which the distorted mesh (133) is relaxed (140) to an applicable state. 

4. A method as claimed in Claim 3, in which prior to step iii) an initial surface (1 30) is fit to the first set of data points, 
and in step iii) the mesh (133) is fit to the initial surface (130). 
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5. A method as claimed in Claim 4, in which the initial surface (130) is an applicable surface, and in which after fitting 
of the mesh (133) to the initial surface (130) at least some of the second set of data points (117) are moved (132) 
closer to corresponding ones of the first set of data (95) points during which the mesh (133) is distorted. 

s 6. A method as claimed in any of Claims 3 to 5, in which data points (1 34) in the second set of data points are not fit 
to the first set of data points (95) if said data points (134) in the second set of data points do not correspond closely 
enough to any of the data points in the first set of data points (95). 

7. A method as claimed in any of Claims 3 to 6 f in which the relaxation of the mesh (140) takes place in an iterative 
10 process in which the second set of data points (117) is adjusted incrementally until distances between points in 

the second set of data points are equalized. 

8. A method as claimed in any of Claims 2 to 7, in which the non-planar applicable surface is a curled document (30). 

is 9. A method as claimed in Claim 8, in which the extent of the document (30) is estimated by fitting a rectangle (124) 
around extreme points (126,128) of the first set of data points (95). 

10. A method as claimed in Claim 8 or Claim 9, in which the image capture means (2) projects a structured light pattern 
(26) that forms separated light stripes (35) across the non-planar applicable surface (30), the first set of data points 

20 (95) being generated from the light stripes (35). 

11. A method as claimed in Claim 10, in which step ii) includes the steps of: 

a) creating a difference image by taking a difference between an image captured with the stripes (33) and an 
25 image captured without the stripes (31); 

b) thresholding the difference image to discard portions below a threshold; 

c) counting detected stripes across the difference image in order to identify individual stripes (35); 

30 

d) triangulating (84) the image of the non-planar surface at points (86) corresponding with identified stripes 
(35) to generate the first set of data points (95). 
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Fig. 3 
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Fig. 10B 
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Fig. 11A 




Fig. 12 
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